Skip to content

typev: add static, upload and echo benchs#725

Merged
MDA2AV merged 3 commits into
MDA2AV:mainfrom
praisethemoon:sch/typev-even-more-bench
May 18, 2026
Merged

typev: add static, upload and echo benchs#725
MDA2AV merged 3 commits into
MDA2AV:mainfrom
praisethemoon:sch/typev-even-more-bench

Conversation

@praisethemoon
Copy link
Copy Markdown
Contributor

Description

adds upload, static and echo-ws* benchmarks for typev.
local bench lite runs.


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run all benchmark tests
/benchmark -f <framework> -t <test> Run a specific test
/benchmark -f <framework> --save Run and save results (updates leaderboard on merge)

Always specify -f <framework>. Results are automatically compared against the current leaderboard.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.

@praisethemoon
Copy link
Copy Markdown
Contributor Author

/benchmark -f typev

@github-actions
Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: typev | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 1,890,757 6579.9% 208MiB +1.0% -17.1%
baseline 4096 1,860,357 6770.2% 412MiB -3.0% +7.3%
pipelined 512 45,838,000 6521.5% 289MiB -4.4% +4.7%
pipelined 4096 46,887,388 6440.2% 452MiB -3.1% +27.3%
limited-conn 512 890,505 4638.0% 226MiB -2.4% +7.1%
limited-conn 4096 1,257,841 6362.6% 333MiB -2.0% +9.5%
json 4096 1,059,398 6720.7% 408MiB -1.7% +1.5%
upload 32 3,069 1958.4% 197MiB NEW NEW
upload 256 3,015 3671.1% 271MiB NEW NEW
static 1024 1,031,804 6431.2% 417MiB NEW NEW
static 4096 1,061,789 6166.0% 574MiB NEW NEW
static 6800 1,066,206 6198.3% 835MiB NEW NEW
echo-ws 512 2,893,471 6355.5% 282MiB NEW NEW
echo-ws 4096 2,950,864 6704.6% 469MiB NEW NEW
echo-ws 16384 2,928,563 6159.7% 1.1GiB NEW NEW
echo-ws-pipeline 512 19,284,998 6476.8% 194MiB NEW NEW
echo-ws-pipeline 4096 19,369,375 6323.7% 466MiB NEW NEW
echo-ws-pipeline 16384 18,771,724 6440.7% 954MiB NEW NEW
Full log
  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    159us    110us    120us    134us   3.12ms

  91384672 frames sent in 5.00s, 91384112 frames received
  Throughput: 18.27M req/s
  Bandwidth:  121.98MB/s
  WS upgrades: 217
  WS frames:   91384112
  Latency samples: 91383840 / 91384112 responses (100.0%)
[info] CPU 6404.8% | Mem 280MiB

=== Best: 19284998 req/s (CPU: 6476.8%, Mem: 194MiB) ===
[info] saved results/echo-ws-pipeline/512/typev.json
httparena-bench-typev
httparena-bench-typev

==============================================
=== typev / echo-ws-pipeline / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    356us    144us    661us    871us   9.05ms

  96581760 frames sent in 5.00s, 96579312 frames received
  Throughput: 19.31M req/s
  Bandwidth:  128.91MB/s
  WS upgrades: 446
  WS frames:   96579312
  Latency samples: 96579248 / 96579312 responses (100.0%)
[info] CPU 6295.5% | Mem 299MiB

[run 2/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    206us    109us    197us    516us   10.60ms

  95420320 frames sent in 5.00s, 95420016 frames received
  Throughput: 19.08M req/s
  Bandwidth:  127.35MB/s
  WS upgrades: 508
  WS frames:   95420016
  Latency samples: 95419808 / 95420016 responses (100.0%)
[info] CPU 6612.7% | Mem 399MiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    205us    107us    180us    418us   12.50ms

  96829184 frames sent in 5.00s, 96846879 frames received
  Throughput: 19.36M req/s
  Bandwidth:  129.27MB/s
  WS upgrades: 446
  WS frames:   96846880
  Latency samples: 96828528 / 96846879 responses (100.0%)
  Latency overflow (>5s): 64

  WARNING: 18446744073709551615/96846879 responses (19047329417512.3%) had unexpected status (expected 2xx)
[info] CPU 6323.7% | Mem 466MiB

=== Best: 19369375 req/s (CPU: 6323.7%, Mem: 466MiB) ===
[info] saved results/echo-ws-pipeline/4096/typev.json
httparena-bench-typev
httparena-bench-typev

==============================================
=== typev / echo-ws-pipeline / 16384c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    439us    114us    624us   3.24ms   12.90ms

  91572464 frames sent in 5.00s, 91567263 frames received
  Throughput: 18.31M req/s
  Bandwidth:  122.23MB/s
  WS upgrades: 543
  WS frames:   91567264
  Latency samples: 91567232 / 91567263 responses (100.0%)
  Reconnects: 463951

  WARNING: 18446744073709551615/91567263 responses (20145566733505.6%) had unexpected status (expected 2xx)
[info] CPU 6352.4% | Mem 466MiB

[run 2/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    189us    107us    121us    204us   9.53ms

  93875232 frames sent in 5.00s, 93858624 frames received
  Throughput: 18.77M req/s
  Bandwidth:  125.30MB/s
  WS upgrades: 1038
  WS frames:   93858624
  Latency samples: 93858624 / 93858624 responses (100.0%)
[info] CPU 6440.7% | Mem 954MiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    310us    114us    221us   2.20ms   20.20ms

  89500528 frames sent in 5.00s, 89486256 frames received
  Throughput: 17.89M req/s
  Bandwidth:  119.47MB/s
  WS upgrades: 892
  WS frames:   89486256
  Latency samples: 89486256 / 89486256 responses (100.0%)
  Reconnects: 498848
[info] CPU 6067.4% | Mem 1.1GiB

=== Best: 18771724 req/s (CPU: 6440.7%, Mem: 954MiB) ===
[info] saved results/echo-ws-pipeline/16384/typev.json
httparena-bench-typev
httparena-bench-typev
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-pipeline-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-pipeline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-pipeline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → powersave

@MDA2AV
Copy link
Copy Markdown
Owner

MDA2AV commented May 18, 2026

/benchmark -f typev --save

@github-actions
Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions
Copy link
Copy Markdown
Contributor

⚠️ /benchmark --save cannot start: main has diverged and cannot be auto-merged into this branch. Please merge or rebase main manually, push, and re-run /benchmark --save.

@praisethemoon
Copy link
Copy Markdown
Contributor Author

/benchmark -f typev --save

@github-actions
Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@praisethemoon
Copy link
Copy Markdown
Contributor Author

Gonna take a look at upload.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: typev | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 1,920,320 6506.2% 209MiB +2.6% -16.7%
baseline 4096 1,920,429 6204.2% 390MiB +0.1% +1.6%
pipelined 512 48,515,708 6431.7% 296MiB +1.2% +7.2%
pipelined 4096 48,124,140 6140.7% 387MiB -0.6% +9.0%
limited-conn 512 881,664 4727.6% 238MiB -3.3% +12.8%
limited-conn 4096 1,299,282 6664.6% 354MiB +1.3% +16.4%
json 4096 1,055,884 6719.4% 404MiB -2.0% +0.5%
upload 32 3,084 1933.4% 192MiB NEW NEW
upload 256 2,999 3858.7% 266MiB NEW NEW
static 1024 973,340 6476.6% 413MiB NEW NEW
static 4096 1,071,937 6300.9% 595MiB NEW NEW
static 6800 1,072,338 6445.8% 1.3GiB NEW NEW
echo-ws 512 2,870,165 6621.8% 270MiB NEW NEW
echo-ws 4096 3,032,878 6184.7% 454MiB NEW NEW
echo-ws 16384 2,916,886 6178.8% 500MiB NEW NEW
echo-ws-pipeline 512 18,569,209 6477.0% 195MiB NEW NEW
echo-ws-pipeline 4096 19,144,768 6613.2% 428MiB NEW NEW
echo-ws-pipeline 16384 19,144,384 6222.4% 1.2GiB NEW NEW
Full log


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    161us    110us    121us    134us   167.50ms

  92253824 frames sent in 5.00s, 92253280 frames received
  Throughput: 18.45M req/s
  Bandwidth:  123.14MB/s
  WS upgrades: 215
  WS frames:   92253280
  Latency samples: 92337696 / 92253280 responses (100.1%)
[info] CPU 6507.2% | Mem 276MiB

=== Best: 18569209 req/s (CPU: 6477.0%, Mem: 195MiB) ===
[info] saved results/echo-ws-pipeline/512/typev.json
httparena-bench-typev
httparena-bench-typev

==============================================
=== typev / echo-ws-pipeline / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    279us    112us    455us    778us   11.80ms

  95664096 frames sent in 5.00s, 95660576 frames received
  Throughput: 19.13M req/s
  Bandwidth:  127.69MB/s
  WS upgrades: 362
  WS frames:   95660576
  Latency samples: 95660448 / 95660576 responses (100.0%)
[info] CPU 6237.0% | Mem 278MiB

[run 2/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    202us    109us    202us    573us   11.00ms

  95724112 frames sent in 5.00s, 95723840 frames received
  Throughput: 19.14M req/s
  Bandwidth:  127.77MB/s
  WS upgrades: 545
  WS frames:   95723840
  Latency samples: 95723567 / 95723840 responses (100.0%)
  Latency overflow (>5s): 32
[info] CPU 6613.2% | Mem 428MiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    215us    110us    212us    888us   10.10ms

  95201488 frames sent in 5.00s, 95201328 frames received
  Throughput: 19.03M req/s
  Bandwidth:  127.06MB/s
  WS upgrades: 548
  WS frames:   95201328
  Latency samples: 95200912 / 95201328 responses (100.0%)
  Latency overflow (>5s): 16
[info] CPU 6254.6% | Mem 491MiB

=== Best: 19144768 req/s (CPU: 6613.2%, Mem: 428MiB) ===
[info] saved results/echo-ws-pipeline/4096/typev.json
httparena-bench-typev
httparena-bench-typev

==============================================
=== typev / echo-ws-pipeline / 16384c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    392us    110us    261us   2.83ms   11.80ms

  94928912 frames sent in 5.00s, 94922816 frames received
  Throughput: 18.98M req/s
  Bandwidth:  126.70MB/s
  WS upgrades: 491
  WS frames:   94922816
  Latency samples: 94922816 / 94922816 responses (100.0%)
  Reconnects: 4366
[info] CPU 6141.3% | Mem 482MiB

[run 2/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    367us    118us    621us   2.16ms   20.50ms

  92239280 frames sent in 5.00s, 92221792 frames received
  Throughput: 18.44M req/s
  Bandwidth:  123.12MB/s
  WS upgrades: 1349
  WS frames:   92221792
  Latency samples: 92221472 / 92221792 responses (100.0%)
  Reconnects: 485856
[info] CPU 6646.2% | Mem 1.0GiB

[run 3/3]
gcannon v0.5.3 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    182us    107us    120us    161us   9.92ms

  95740512 frames sent in 5.00s, 95721920 frames received
  Throughput: 19.14M req/s
  Bandwidth:  127.79MB/s
  WS upgrades: 1162
  WS frames:   95721920
  Latency samples: 95721920 / 95721920 responses (100.0%)
  Reconnects: 4093
[info] CPU 6222.4% | Mem 1.2GiB

=== Best: 19144384 req/s (CPU: 6222.4%, Mem: 1.2GiB) ===
[info] saved results/echo-ws-pipeline/16384/typev.json
httparena-bench-typev
httparena-bench-typev
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-pipeline-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-pipeline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/echo-ws-pipeline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → powersave

@MDA2AV MDA2AV merged commit 549dfa8 into MDA2AV:main May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants